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validity  of  thp  current  MACS  training  programs.  In  general,  the  correlational 
approach  indicated  low  MACS  reliability  (correlations  of  MACS  scores  with  MACS 
scores),  low  MACS  external  validity  (correlations  of  MACS  performance  with 
live-fire  performance),  and  generally  acceptable  live-fire  reliability  (corre¬ 
lations  of  live-fire  measures  with  live-fire  measures).  The  most  significant 
finding  was  that  MACS  measures  of  var iabll ity  of  light-pen  readings  (  a  measure 
of  how  steady  the  weapon  was  held)  showed  highly  significant  reliability  and 
, generally  acceptable  external  validity.  This  was  in  sharp  contrast  to  the  MACS 
point-of-aim  scores.  Two  hypotheses  are  proposed  for  future  MACS  development: 
that  the  MACS  software  be  Changed  in  a  way  that  slight  movements  of  the  weapon 
during  trigger  manipulation  be  included  in  the  determination  of  the  MACS 
point-of-aim  scores;  and,  that  measures  of  variability  be  exploited  in  the  MACS 
-.oftware  in  order  to  provide  prediction  of  marksmanship  performance,  feedback 
for  training  marksmanship,  and  diagnosis  and  remediation  of  marksmanship 
problems. 
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INTRODUCTION 


The  Multipurpose  Arcade  Combat  Simulator  (MACS)  la  a  low  cost,  part-task 
simulator/ trainer. being  developed  by  the  US  Army  Research  Institute's  Fort 
Banning  Field  Unit .'''>Using  off-the-shelf  components,  I’lACS  provides  the  shooter 
with  valuable  training  feedback  about  location  of  bullet  strike  at  various 
ranges,  effects  of  wind  and  gravity,  etc.  The  key  component  in  MACS  is  a 
specially  designed  light  pen  which  can  be  removed  from  one  weapon  system  (e.g., 
the  M16A1  rifle)  and  attached  to  another  weapon  system  (e.g.,'  the  M203  grenade 
launcher  or  M72A2  light  antitank  weapon).  Hence,  MACS  is  a  trainer  which  can 
train  a  variety  of  tasks  on  a  variety  of  weapon  systems, ^see  Figure  1).  For  a 
more  detailed  description  of  MACS,  see  Schroeder  (1983a)7''; 

■Recently,  preliminary  experimental  evaluations  of  the  MACS  M16A1  rifle, 

M203  grenade  launcher,  and  M72A2  light  antitank  weapon  weia  conducted, (see 
Perkins,  Selby,  Broom,  &  Osborne,  1984).  ■  The  results  of  .  the  sc  ~eva luat ions 
showed  positive,  transfer  to  live  fire  for  the  grenade  launcher  and  LAW,  but 
minimal  transfer  for  the  MACS  ti.iie  application.  Although  there  were  some 
problems  encor.ntered  in  the  field  evaluation  (see  Perkins  et  al.  for  details)^ 
the  lack  of  significant  posit  -a  transfer  was  surprising  given  the  extensive 
concurrent  training  provided  on  Perkins  et  al.  report  concentrated  on 

the  experimentel-resHlreef'  Hou .  ^there  was  little  correlational  analysis 
discussed  and  such  analyses  could  prove  useful  in  discovering  new  directions, 
corrections,  and  software  improvrticats  for  MACS.  The  purpose  of  this  report  is 
to  provide  additional  correlational  analyses  in  an  attempt  to  identify  promising 
new  avenues  of  research  and  developffi.»nt  for  the  MACS  system. 


IROCEDURE 


In  general,  the  experimental  approach  of  Perkins  et  al.  was  to  provide 
concurrent  training  on  MACS  for  one  platoon  (experimental  group)  and  no  MACS 
concurrent  training  for  a  second  platoon  (control  group).  The  subsequent 
analyses  focused  on  comparisons  between  live-fire  scores  for  the  two  groups. 

The  correlational  analyses  in  the  present  paper  concentrate  on  the  experimental 
group  only  because  a  vast  amount  of  MACS  performance  data  was  collected  on  those 
subjects.  Figure  2  (from  Perkins  et  al.)  shows  the  various  Basic  Rifle 
Marksmanship  (BR.M)  periods  of  instruction  and  corresponding  MACS  training 
programs  which  were  presented  to  the  experimental  subjects  during  concurrent 
training  for  those  live-fire  periods  (see  Perkins  et  al.  for  descriptions  of  the 
training  in  each  of  the  BRM  and  ;-lACS  programs). 

The  following  results  and  discussion  sections  will  cover  three  different 
topics:  the  correlations  of  MACS  measures  with  otner  MACS  measures,  the 
correlations  live-fire  measures  with  other  live-fire  measures,  and  the 
correlations  of  MACS  measures  with  live-fire  measures.  By  analyzing  the 
correlations  of  MACS  measures  with  other  MACS  measures,  the  reliability  of  MACS 
can  be  assessed.  Similarly,  by  addressing  the  correlations  of  live-fire 
measures  with  other  live-fire  measures,  the  correlational  ’’celling'  can  be, 
identified  (i.e.,  how  can  a  simulator  predict  live  fire  better  than  an  identical 
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FIGURE  2.  BRM  training  periods  and  corresponding 
MACS  training  programs  presented  during 
concurrent  training. 


live-fire  Cask).  By  analyzing  the  correlations  of  MACS  with  live  fire,  the 
strergchs  and  weaknesses  of  MACS  can  be  identified.  More  specifically,  in  the 
development  of  a  simulator/ trainer ,  the  first  goal  should  be  to  identify 
simulator  measures  which  correlate  with  the  measures  that  the  simulator  is 
trying  to  train  (l.e.,  live-fire  marksmanship  scores  in  this  case).  The 
correlational  approach  provided  in  the  present  paper  provides  a  development  aid 
by  identifying  the  MACS  variables  with  the  highest  potential.  Once  promising 
simulator  measr.res/programs  are  identified,  then  efforts  can  be  concentrated  to 
Improve  chose  measures  while  less  promising  measures/programs  can  be  discarded. 


RESULTS 


The  Correlation  of  MACS  Measures  with  MACS  Measures 


For  the  purpose  of  this  analysis,  nine  independent  measures  of  MACS 
performance  were  included  from  all  stages  of  MACS  training  (see  Figure  2).  A 
list  of  these  nine  measures  along  with  a  brief  explanation  of  each  is  presented 
in  Table  1.  Table  2  shows  the  resulting  correlation  matrix.  There  are  three 
general  classes  of  measurement  shown  in  Table  2:  the  amount  of  time  it  took 
soldiers  to  fire  at  targets  in  the  early  pretest  and  posttest  (Pretest  Latency 
and  Posttest  Latency);  the  variability  of  weapon  polnt-of-aim  prior  to  firing 
(Pretest  Steady  Position  and  Posttest  Steady  Position);  and  accuracy  scores 
(Down-Range  Feedback,  Field  Fire  I,  Field  Fire  II,  and  Record  Fire).  The 
latency  measures  did  not  correlate  significantly  with  each  other  indicating  very 
low  reliability.  In  addition,  the  latency  measures  did  not  correlate 
«lgnif icancly  with  the  other  measures  with  the  exception  of  a  significant 
correlation  between  Pretest  Latency  with  Field  Fire  II  scores  (£  <.05 ).  There 
was  high  test-retest  reliability  for  the  fwo  variability  scores  (£<.001),  but 
low  correlations  between  those  steady  position  scores  and  other  MACS  measures 
(except  for  the  significant  correlation  between  Posttest  Steady  Position  and 
Field  Fire  II  scores  (ji^<.05).  Finally,  there  were  no  significant  correlations 
among  the  v'>.rlous  accuracy  measures  with  the  exception  of  Field  Fire  II  and  MACS 
Record  Fire  (^  <.01 ). 

In  summary,  the  correlational  analysis  indicated  a  general  lack  of 
reliability  among  the  three  general  classes  of  MACS  measures.  In  addition,  only 
the  variability  measures  (Steady  Postltion  scores)  showed  significant 
within-class  reliability. 


The  Correlations  of  Live-Fire  Measures  with  Live-Fire  Measures 


Table  3  provides  a  list  of  the  various  live-fire  measurements  included  in 
the  current  analysis.  One  measure  was  included  from  each  period  of  instruction 
from  Period  4  to  Period  10  and  two  measures  were  included  for  Period  11  (Combat 
Fire  and  Record  Fire).  All  measures  used  were  obtained  directly  from  standard 
company  performance  records  except  Shot-Group  Size  from  Period  6.  Shot-Group 
Size  from  Period  6  was  calculated  from  actual  target  measurements  and  represents 
the  mean  radial  distance  from  the  center  of  the  shot  group'  to  the  various  shot 
locations.  Tdble  4  shows  the  correlation  matrix  within  the  various  live-fire 
measures.  The  Record  Fire  score  correlated  significantly  with  all  other 
measures  except  Number  of  Rounds  to  Zero  in  Period  4.  The  lack  of  correlation 


Table  1 


Variable 
Pretest  Latency 

Fosttest  Latency 

Pretest  Steady  Position 

Posttest  Steady  Position 

Down-Range  Feedback 

Field  Fire  I 

Field  Fire  II 

MACS  Record  Fire 


MACS  Variables  and  Descriptions 


Description 

Median  time  to  fire  the  weapon  in  MACS  Aiming 
pretest. 

Median  time  to  fire  the  weapon  In  MACS  Aiming 
posttest. 

Mean  sum  of  standard  deviations  for  X  and  Y 
for  5  light-pen  readings  before  6  shots 
In  MACS  Aiming  pretest. 

Mean  sum  of  standard  deviations  for  X  and  Y 
for  5  light-pen  readings  before  6  shots 
In  MACS  Aiming  posttest. 

Mean  median  score  on  5  exposures  to  MACS  Down-Range 
Feedback  program. 

Mean  median  score  on  2  exposures  to  MACS  Field 
Fire  I  program. 

Mean  median  score  on  2  exposures  to  MACS  Field 
Fire  II  program. 

Mean  median  score  on  2  exposures  to  MACS  Record 
Fire  program. 
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Correlation  Matrix?  MACS  With  MACS 
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Table  3 


Llve-Flre  Variables  and  Descriptions 


Variable 

Rounds  to  Zero,  Period  4 
Hits  in  Period  5 

Shot-Group  Size,  Period  6 

lilts  in  Period  7 

Hits  in  Period  8 

Hits  in  Period  9 

Practice  Record  Fire 

Combat  Fire 
Record  Fire 


Description 

Number  of  rounds  to  zero  in  Period  4. 

Number  of  hits  on  scaled  silhouette  targets 
in  Period  5. 

Mean  radial  error  from  shot-group  center 
on  the  75  m  target  in  Period  6. 

Number  of  hits  on  field-fire  killable 
targets  in  Period  7. 

Number  of  hits  on  field-fire  kiliable 
targets  in  Period  ,8. 

Number  of  hits  on  time. J-f ire  scaled 
silhouette  targets  in  Period  9. 

Number  of  hits  in  Practice  Record  Fire 
(Period  10). 

Number  of  hits  in  Combat  Fire  (Period  11a). 

Number  of  hits  in  Qualification  Record  Fire 
(Period  11b). 
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between  Period  4  scores  and  Record  Fire  scores  is  not  surprising  given  Che 
amount  of  time  that  elapsed  between  Che  two  measurements  and  the  large  number  of 
uncontrollable  variables  which  can  influence  Che  Period  4  measure  (e.g.,  lack  of 
familiarity  with  the  weapon,  bad  initial  sight  settings  which  throw  rounds 
completely  off  Che  paper,  etc.).  According  to  the  present  results,  the  best 
single  predictor  of  Record  Fire  score  was  Practice  Record  Fire  followed  by 
Shot-Group  Size  in  Period  6,  Combat  Fire,  and  Chen  a  cluster  of  measures  which 
correlated  about  Che  same  (Hits  in  Period  5,  Kits  in  Period  9,  Hits  in  Period  7, 
and  Hits  in  Period  8).  In  terms  of  magnitude  of  correlation  coefficient,  Che 
best  single  early  predictor  of  Record  Fire  was  Shot-Croup  Size  in  Period  6. 
However;,  since  Hits  In  Period  3  can  be  obtained  a  day  earlier  and  with  less 
difficulty,  it  may  be  the  most  functional  eatly  predictor  of  Record  Fire  scores. 

In  conclusion,  Che  inter-correlations  of  various  live-fire  scores  was 
generally  high.  Although  there  were  not  consistently  significant  correlations 
among  the  various  periods  of  instruction  leading  to  record  fire»  there  were 
statistically  significant  correlations  between  all  live-fire  periods  and  record 
fire  except  for  the  number  of  rounds  to  zero  in  period  4. 


The  Correlations  of  MACS  measures  with  Live-Fire  Measures 


Table  3  shows  the  correlations  betwe'.-n  the  various  MACS  measures  and  Che 
various  live-fire  measures.  There  was  a  general  lack  of  correlation  between  tne 
various  MACS  scores  and  llve-fire  scores.  With  the  exception  of  a  significant 
correlation  between  the  MACS  Field  Fire  I  scores  and  the  Shot-Group  Size  in 
Period  6,  the  only  significant  correlations  involved  the  MACS  measures  of 
variability  (Posttest  Steady  Position  with  Rounds  to  Zero  in  Period  4,  Pretest 
Steady  Position  with  Hits  in  Period  7,  Posttest  Steady  Position  with  Hits  in 
Period  7,  Pretest  Steady  Position  with  Practice  Record  Fire,  Posttest  Steady 
Position  with  Practice  Record  Fire,  and  Posttest  Steady  Position  with  Record 
Fire). 


In  conclusion,  the  correlations  among  MACS  scores  and  llve-fire  scores  were 
generally  low.  The  exceptions  were  the  two  MACS  measure  of ,  variability  of 
light-pen  readings  (measuring  how  steady  the  weapon  was  held  just  prior  to 
shooting).  In  those  two  measures  (Pretest  Steady  Position  and  Posttest  Steady 
Position),  6  of  18  correlations  were  statistically  significant. 


DISCUSSION 


In  general,  Che  JIACS  system  was  found  to  be  low  in  reliability  (as  measured 
by  correlations  of  MACS  measures  with  other  MACS  measures)  auu  low  in  external 
validity  (as  measured  by  correlations  of  >JiCS  measures  with  live-fire  measures). 
In  comparison,  there  were  generally  acceptable  correlations  among  the  various 
live-fire  measures. 

On  the  surface,  these  results  seem  to  indicate  that  not  only  can't  MACS 
train  marksmanship  skills  (i.e.,  Perkins  et  al.)  but  in  addition,  MACS  can't 
predict  live-fire  performance.  However,  there  was  one  potentialfy  interesting 
trend  in  the  present  data.  More  specifically,  the  high  reliability  shown  in  the 
variability  measures  (as  opposed  to  point-of-aim  measures)  and  the  external 
validity  shown  in  the  significant  correlations  of  those  variability  measures 
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Indicates  P  ^  .OS 
Indicates  P  <  .01 
Indicates  P  <  .001 


with  5  of  the  live-fire  measures  incicste  a  promising  new  measure.  Is  it 
possible  chat  poinc-of-aim  measures  of  accuracy  are  not  Che  best  predictors  of 
marksmanship  in  Che  MACS  system?  Accuracy  measures  were  used  in  the  original 
evaluation  (Perkins  et  al.)  because  they  represented  the  most  intuitive  and 
traditional  measure  of  performance  in  a  marksmanship  cask.  However,  perhaps 
electronic  simulators  like  MACS  provide  additional  and  more  powerful  measures  of 
marksmanship  ability  which  are  not  utilized  because  they  have  not  been 
traditionally  available. 

There  is  another  interesting  feature  in  the  present  data.  Although  this 
study  was  Che  first  time  chat  MACS  has  been  correlated  with  live-fire,  in  an 
earlier  study  (Schroeder,  1983b)  MACS  oolnt-of-alm  scores  were  found  to 
significantly  correlate  with  scores  on  another  marksmanship  simulator  —  Che 
Moving  Target  Rifle  Marksmanship  Trainer  (tfTRMT).  Other  research  has  shown  chat 
HTRMT  scores  correlate  significantly  with  live  fire  (Schendel  and  Heller,  1982; 
and  Schendel,  Heller,  Finley,  and  Hawley,  1983).  This  trend  of  results  is 
incongruous  with  the  generally  low  correlations  of  MACS  poinc-of-aim  scores  with 
live-fire  scores  found  in  the  present  study.  This  apparant  inconsistency  caused 
the  present  author  to  review  Che  MACS  development  effort  to  determine  if  any 
hardware  and/or  software  changes  could  have  resulted  in  the  discrepancy  between 
early  MACS  results  and  the  present  results.  Hardware  changes  were  unlikely  to 
have  caused  the  discrepancy  since  the  reliability  and  accurracy  of  Che  light  pen 
had  actually  been  Improved  over  the  development  effort.  However,  there,  was  a 
seemingly  insignificant  software  change  chat  may  have  resulted  in  lower 
correlations  in  Che  present  study.  The  change  involves  Che  MACS  software  and 
how  it  determines  point-of-aim.  In  the  original  MACS  software  that  the  present 
author  wrote,  5  light-pen  readings  were  collected  after  tite  trigger-switch  was 
closed,  since  the  software  can  provide  readings  at  Che  rate  of  about  20  per 
sec,  this  represented  a  measurement  window  of  about  .23  sec.  After  Che  3 
light-pen  readings  were  taken,  both  "X"  readings  (horizontal)  and  “Y"  readings 
(vertical)  were  sorted  and  Che  medians  were  determined.  This  approach  was  taken 
to  stabilize  the  light-pen  readings  and  eliminate  any  potential  deviant  scores. 
It  was  this  procedure  Chat  was  used  in  the  study  which  found  significant 
correlations  with  the  MTRMT.  Subsequently,  software  contractors  were  brought  in 
to  make  improvements  in  the  software.  The  present  author  directed  the  software 
developers  Co  change  Che  above  procedure  in  order  to  increase  Che  speed  of 
operation  of  the  MACS  system.  In  the  new  procedure,  the  light  pen  continuously 
took  readings  and  updated  a  fixed  stack.  When  the  trigger-switch  was  ctosed, 
the  last  5  X  and  Y  readings  before  trigger-pull  were  sorted  and  medians 
calculated;  The  two  procedures  were  functionally  the  same,  except  for  the  time 
window.  Could  a  difference  of'  .23  sec  in  the  time  window  have  resulted  in  the 
apparently  large  difference  in  results?  Perhaps  the  answer  is  "yes”  if  small 
changes  in  the  position  of  Che  weapon  due  to  trigger  manipulation  affect  both 
point-of-aim  scores  on  a  simulator  and  in  live  fire.  More  specifically,  one  of 
the  most  important  principles  in  marksmanship  is  that  the  trigger  should  be 
squeezed  because  rapid  trigger  manipulation  typically  results  in  movement  of  the 
weapon  and  error  in  the  subsequent  shot.  Using  the  5  readings  before  trigger 
manipulation  is  less  likely  to  pick  up  distortions  in  the  point-of-aim  than 
using  the  3  readings  after  trigger  manipulation  for  those  poor  shooters  who 
quickly  "jerk"  the  trigger.  This  is  especially  true  given  the  median  routine  in 
the  software  that  presumably  sorts  out  deviant  scores  due  to  the  light  pen,  but 
in  this  case,  may  be  sorting  out  deviant  scores  that  are  due  to  shooting  errors. 
Because  this  process  would  make  poor  shooters  appear  better  than  they  really 
are,  the  range  of  the  MACS  scores  would  be  restricted  and  the  magnitudes  of  the 
correlations  would  be  limited.  In  addition,  this  may  explain  why  such  a  high 
percentage  of  the  experimental  MACS  subjects  in  Perkins  et  al.  scored  very  high 


on  the  MACS  aystem  (l.e..  If  holding  the  weapon  steady  during  trigger 
aanipulation  is  a  key  coaponeht  to  marksm4inship  and  the  second  generation  liACS 
software  was  nut  looking  at  that  compone-t,  then  perhaps  MACS  was  measuring  the 
ability  to  achieve  a  proper  initial  aiming  point  —  a  fairly  easy  task). 

The  two  hypotheses  discussed  above  suggest  an  interesting  redirection  for 
software  development  of  the  MACS  system.  First,  studies  need  to  be  done  to 
determine  the  optimum  time  window  and  statistical  method  for  determining 
polnt-of-aim.  This  should  be  done  using  correlation  with  live  fire  as  a 
criterion.  Second,  the  high  test-retest  reliability  shown  in  the  variability 
measure  reflecting  steadiness  in  the  weapon  should  be  investigated  as  a  possible 
meaningful  measure  for  both  the  prediction  of  marksmanship  ability  and  as  a 
feedback  variable  for  training  marksmanship.  In  general,  what  is  suggested  by 
the  present  author  is  a  correlational  time-series  type  analysis  that  empirically 
determines  the  best  w'.ndow(s)  and  statistical  method(3)  for:  predicting 
marksmanship,  diagnosing  and  remediating  marksmanship  problems,  and  providing 
feedback  for  marksmanship  training.  The  current  expertise  in  marksmanship 
training  should  help  guide  this  development  effort.  For  example,  in  the  Army's 
Basic  Rifle  Marksmanship  Program  of  Instruction,  the  four  fundamentals  are: 
steady  position,  trigger  squi'eze,  aiming,  and  breath  control.  If  these  are 
indeed  fundamentals  of  marksmanship,  then  why  not  design  a  simulator/trainer 
that  measures  these  fo.  components  (or  as  many  as  feasible).  The  new 
redirection  in  MACS  software  development  will  be  to  establish  time  windows  and 
statistical  methods  which  will  hopefully  measure  at  least  three  of  these  four 
areas:  steady  position,  trigger  squeeze,  and  aiming.  Measures  of  variance  for 
selected  time  windows  should  provide  diagnostic,  remedial,  predictive,  and 
training  information  for  the  steady  position  and  trigger  squeeze  variables. 
Measures  of  central  tendency  for  a  selected  time  window  should  provide 
diagnostic,  remedial,  predictive,  and  training  Information  fOr  the  aiming 
variable. 

In  the  near  future,  research  will  be  conducted  which  will  identify  the 
optimal  windows  and  statistical  measures.  ’  The  next  generation  of  MACS  software 
will  incorporate  this  informa'  on.  Different  programs  will  be  written  for  the 
purpose  of  prediction  of  marksu-anship  ability,  diagnosis  and  remediation  of 
marksmanship  problems,  and  for  the  general  training  of  marksmanship  skills  on 
the  M16A1  rifle  and  other  MACS  weapons. 
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